Rainbow between trees (naive)
Rainbow between trees (poisson blend with mixed gradients)
11-747 Learning-based Image Synthesis Manuel Rodriguez Ladron de Guevara
Rainbow between trees (naive)
Rainbow between trees (poisson blend with mixed gradients)
The goal of this assignment is to explore gradient-domain processing, a technique with a broad set of
applications such as blending, tone-mapping, and non-photorealistic rendering. We show some results on Poisson
blending with and without mixed gradients, color2gray, and color transfer.
Poisson blending is a technique that allows to blend two images seamlessly, utilizing the gradients of both ends, source and target images. That is, given a source image (image you want to paste), a target image (image where you will paste on), and a mask (representing the portion of the source image that you want to paste), the Poisson blending algorithm will preserve the gradient of the source region without changing the background pixels. A common downside of this algorithm is that the color can change since we are deliberately ignoring the overall intensity!
Let's first understand the gradients of an image. Let's consider a 1d grayscale array of 5 pixels. Consider the
following barchart:
In this plot, pixel 1 has an intensity of 2 $p_1=2$, pixel 3 an intensity of 3 $p_3=3$, and so on. The chart
corresponds to the image below:
Each pixel's gradient is its derivative. How do we calculate the derivative of a pixel? Well, the definition of a derivative is the rate of change of a function given a direction and a step. Here, we can consider the direction to the right, and the step is 1. So that we have: $$\frac{p_{3+1} - p_3}{1} = p_4 - p_3 = 2$$ In this way, we can store pixels with their gradients instead of their pixel value. We are interested in gradients since we want to ensure a nice smooth transition between the pasted element and the background image. We can see how each new pixel would have an equation based on gradients with respect to their neighboring pixels, and we can resolve the transition with a system of equations. In its simplest form: $$ p_2 - p_1 = 5\\ p_3 - p_2 = -4 \\ p_4 - p_3 = 2 \\ p_5 - p_4 = -4 \\ p_5 = 1 $$ We solve for least squares as $Ap = b$. In the algorithm, we want to put all unknown variables in the LHS, and the known variables in the RHS. That is, we set the matrix A to store all unknown values $p$ that we want to solve for, and the values we know are stored in a vector $b$.
Before we jump into the Poisson blending algorithm, we are asked to resolve a toy problem. We are computing the x and y gradients from an image s, to use all the gradients, plus one pixel intensity, to reconstruct an image v. Let's denote the intensity of the source image at $(x,y)$ as $s(x,y)$ and the values of the image to solve for as $v(x,y)$. For each pixel, we have two objectives:
Starry night. Background image (target).
The Scream. Source image.
Mask of desired region in source, aligned to the target.
In order to make a seamless transition between any two images we need to be specific about the parts involved. Let the image we want to have as background target, and the image from which we will crop a portion to paste into the background source. We first need to create a mask to cover in black all those parts from the source that we don't want, and align this mask to the target image. We use a given code to do the process of cropping, creating mask and aligning. Based on this, the Poisson formula is given by: $$\boldsymbol{v} = argmin_{\boldsymbol{v}} \sum_{i\in S, j\in N_i \cap S}((v_i - v_j) - (s_i - s_j))^2 + \sum_{i \in S, j \in N_i \cap\neg S}((v_i - t_j) - (s_i - s_j))^2$$ where $S$ corresponds to the masked region, $i$ the current pixel, $j$ the neighboring pixel, $v$ the pixel we are trying to solve, $s$ the pixel at the source image, and $t$ he pixel at the target image.
We are going to show some interesting, not so interesting, and failure results using this technique.
Source.
Target
Naive copy-paste.
Poisson blending
This example is a somewhat successful example of the Poisson blending algorithm. There is a nice improvement from the naive copy-paste to the result of the algorithm. This works nicely because the texture around the cropped eye is similar to the target image, although the intensities are quite different. It is not realistic because of the size of the eye, so it could have been better.
Source.
Target
Naive copy-paste.
Poisson blending
This example is a another successful example of the Poisson blending algorithm. The interesting part is that it behaves almost like a color transfer algorithm (we will show more of that later). What I like about this particular piece is the mix of styles between Van Gogh and Munch!
Source.
Target
Naive copy-paste.
Poisson blending
This example is a failure. We see how there is a smudge border all along the graffiti we wanted to insert. It creates a strong artifact at the top of the graffiti and the two colors of the wall do not blend properly. One of the reasons of the failure is that the nature of this blending asks for transparency, something that the Poisson blending algorithm can't do (but we will see how to fix that problem later).
Source.
Target
Naive copy-paste.
Poisson blending
This example works nicely, although we also see some blurriness above the penguin's head.
Source.
Target
Naive copy-paste.
Poisson blending
This is a terrible example (sadly, I thought it was going to work!). It seems that Pixel does not want to be in the beach! Jokes aside, this is a clear example that when the source image has a very different background to the target image, in color, intensity and texture, the Poisson blending is not possible.
We introduce various improvements over our baseline.
Mixed Gradients This presents and improvement over the Poisson blending algorithm for images where transparency is important. We use the larger magnitude gradient rather than the source gradient: $$ \boldsymbol{v} = argmin_{\boldsymbol{v}} \sum_{i \in S, j \in N_i \cap S} ((v_i - v_j) - d_{ij})^2 + \sum_{i \in S, j \in N_i \cap \neg S}((v_i - t_j) - d_{ij})^2. $$ where $d_{ij}$ is the value of the larger magnitude between the source and the target image.
Some results using mixed gradients:
Naive copy-paste
Poisson blending
Poisson blending with mixed gradients
Mixed gradients works very well in this example. We see how Poisson blending alone is unable to fuse the two images together. Mixed gradients, however, keep some of the gradient of the target image in the mask area, making this fusion almost spotless.
Naive copy-paste
Poisson blending
Poisson blending with mixed gradients
Another example that shows the advantage of mixed gradients. In cases like this, in which we just need to copy the pixels that are in some certain color (the written part of the source image), the mixed gradients works excellent!
Naive copy-paste
Poisson blending
Poisson blending with mixed gradients
This example shows some interesting results. Poisson blending does a relatively good job on blending the face, although the skin color of the source image makes it very unrealistic and does have a fake look. However the bear seems nicely integrated. The mixed gradients version, does a better job at preserving monalisa's skin color, integrating the moustache and nose nicely while preserving the original face's rounded shape. However, the eyes have a very creepy look as it is mixing the eyes from both sources.
Naive copy-paste
Poisson blending
Poisson blending with mixed gradients
This demonstrates when not to do mixed gradients. For solid images like this, we want to preserve the gradient of the source image, rather than the target.
Color2Gray This gradient-based algorithm creates a gray image with similar intensity than a normal grayscale image but with similar contrast to the original RGB image. Good algorithm for color-blind images.
Original image
Normal color2gray
Gradient-based color2gray
Original image
Normal color2gray
Gradient-based color2gray
Original image
Normal color2gray
Gradient-based color2gray
Color Transfer We implement automatic color transfer from source to target. This simple algorithm achieves nice results and it is very fast to compute.
Example using New York's skyline with 2 different sources
Target
Source 1
Source 2
Result from source 1
Result from source 2
Example in a rural setting
Target
Source
Result